Patient-Specific Preoperative Planning Simulation Techniques

ABSTRACT

A preoperative surgical planning system uses a head-mounted device to execute a preoperative surgical simulation whereby a virtual tool and a virtual anatomical model are provided on the display of the head-mounted device. The virtual tool is tracked relative to the virtual anatomical model in the preoperative surgical simulation in which the virtual tool is moveable in response to receipt of a control input from the wearer of the head-mounted device and wherein the virtual tool is configured to remove a portion of the virtual anatomical model. A planning parameter is automatically generated based on tracking of the virtual tool relative to the virtual anatomical model in the preoperative surgical simulation. The generated planning parameter is stored for future retrieval by a surgical system to facilitate intraoperative surgery based on the generated planning parameter.

CROSS-REFERENCE TO RELATED APPLICATIONS

The subject application is a continuation of U.S. Nonprovisional patentapplication Ser. No. 16/186,979, filed on Nov. 12, 2018, which claimsthe benefit of and priority to U.S. Provisional Patent App. No.62/585,789, filed Nov. 14, 2017, the disclosure of each of theaforementioned applications being hereby incorporated by reference intheir respective entirety.

TECHNICAL FIELD

The disclosure relates generally to surgical systems, and morespecifically, to systems and methods for simulating a surgical procedureto facilitate a later performance of the surgical procedure.

BACKGROUND

Surgical navigation systems assist users in locating objects in one ormore coordinate systems. Surgical navigation systems may employ lightsignals, sound waves, magnetic fields, radio frequency signals, etc. inorder to track positions and/or orientations of the objects. Often thesurgical navigation system includes tracking devices attached to theobjects being tracked. A surgical navigation localizer cooperates withthe tracking devices to ultimately determine positions and/ororientations of the objects. The surgical navigation system monitorsmovement of the objects via the tracking devices.

Surgeries in which surgical navigation systems are used includeneurosurgery and orthopedic surgery, among others. Typically, surgicaltools and anatomy being treated are tracked together in real-time in acommon coordinate system with their relative positions and/ororientations shown on a display. In some cases, this visualization mayinclude computer-generated images of the surgical tools and/or theanatomy displayed in conjunction with real video images of the surgicaltools and/or the anatomy to provide mixed reality visualization. Thisvisualization assists surgeons in performing the surgery.

Even with the assistance of such surgical navigation systems, surgeonsmay still encounter difficulty in performing surgical procedures. Thisis especially true where surgeons are called to perform unfamiliarprocedures or procedures that they have not performed recently.

SUMMARY

According to a first aspect, a preoperative surgical planning system isprovided that comprises: a head-mounted device comprising one or morecontrollers and a display that is configured to be located in front ofthe eyes of a wearer of the head-mounted device, the head-mounted devicebeing configured to receive a control input from the wearer, and the oneor more controllers being configured to execute a preoperative surgicalsimulation with the head-mounted device wherein the one or morecontrollers are configured to: load a virtual tool and a virtualanatomical model for the preoperative surgical simulation; display thevirtual tool and the virtual anatomical model on the display of thehead-mounted device; track the virtual tool relative to the virtualanatomical model in the preoperative surgical simulation in which thevirtual tool is moveable in response to receipt of the control inputfrom the wearer of the head-mounted device and wherein the virtual toolis configured to remove a portion of the virtual anatomical model;automatically generate a planning parameter based on tracking of thevirtual tool relative to the virtual anatomical model in thepreoperative surgical simulation; and store the generated planningparameter for future retrieval by a surgical system to facilitateintraoperative surgery based on the generated planning parameter.

According to a second aspect, a method of operating a preoperativesurgical planning system is provided, the preoperative surgical planningsystem comprising a head-mounted device including one or morecontrollers and a display that is configured to be located in front ofthe eyes of a wearer of the head-mounted device, the head-mounted devicebeing configured to receive a control input from the wearer, and the oneor more controllers being configured to execute a preoperative surgicalsimulation with the head-mounted device, wherein the method comprisesthe one or more controllers performing the following steps: loading avirtual tool and a virtual anatomical model for the preoperativesurgical simulation; displaying the virtual tool and the virtualanatomical model on the display of the head-mounted device; tracking thevirtual tool relative to the virtual anatomical model in thepreoperative surgical simulation whereby the virtual tool moves inresponse to receiving the control input from the wearer of thehead-mounted device and wherein the virtual tool is moved for removing aportion of the virtual anatomical model; automatically generating aplanning parameter based on tracking of the virtual tool relative to thevirtual anatomical model in the preoperative surgical simulation; andstoring the generated planning parameter for future retrieval by asurgical system to facilitate intraoperative surgery based on thegenerated planning parameter.

According to a third aspect, a system is provided that comprises: apreoperative surgical planning system comprising: a head-mounted devicecomprising one or more controllers and a display that is configured tobe located in front of the eyes of a wearer of the head-mounted device,the head-mounted device being configured to receive a control input fromthe wearer, and the one or more controllers being configured to executea preoperative surgical simulation with the head-mounted device whereinthe one or more controllers are configured to: load a virtual tool and avirtual anatomical model for the preoperative surgical simulation, thevirtual anatomical model corresponding to a physical anatomy to betreated; display the virtual tool and the virtual anatomical model onthe display of the head-mounted device; track the virtual tool relativeto the virtual anatomical model in the preoperative surgical simulationin which the virtual tool is moveable in response to receipt of thecontrol input from the wearer of the head-mounted device and wherein thevirtual tool is configured to remove a portion of the virtual anatomicalmodel; automatically generate a planning parameter based on tracking ofthe virtual tool relative to the virtual anatomical model in thepreoperative surgical simulation; and store the generated planningparameter; and an intraoperative surgical system comprising: a roboticmanipulator being configured to support a surgical tool, the surgicaltool being configured to remove material from the physical anatomy to betreated and the surgical tool corresponding to the virtual tool used inthe preoperative surgical simulation, and wherein one or moremanipulator controllers are coupled to the robotic manipulator, the oneor more manipulator controllers being configured to: load the generatedplanning parameter that was automatically generated from thepreoperative surgical simulation; and control the robotic manipulatorbased on the generated planning parameter to remove material from thephysical anatomy with the surgical tool.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages of the present invention will be readily appreciated as thesame becomes better understood by reference to the following detaileddescription when considered in connection with the accompanying drawingswherein:

FIG. 1 is a perspective view of a robotic system.

FIG. 2 is a schematic view of a control system.

FIG. 3 is an illustration of various transforms used in navigation.

FIG. 4 is a block diagram of a surgical planning program.

FIGS. 5A and 5B are perspective views of a preoperative simulation of asurgical procedure using a physical tool and a physical model of apatient's anatomy.

FIG. 6 is a perspective view of a preoperative simulation of a surgicalprocedure using a virtual tool and a virtual model of the patient'sanatomy.

FIG. 7 is a perspective view of a preoperative simulation of a surgicalprocedure using a physical tool with a virtual model of the patient'sanatomy.

FIG. 8 is a perspective view of a patient's anatomy during execution ofan intraoperative surgical procedure.

FIG. 9 is a flowchart of a method of executing a surgical procedure.

DETAILED DESCRIPTION

I. System Overview

In the embodiments disclosed herein, a surgical system is described thatincludes a navigation system and a robotic manipulator to which asurgical tool is removably attached. Alternatively, the system may notinclude the robotic manipulator such that the surgical tool is ahandheld tool usable by a surgeon. At least some portions of thesurgical system may be used to preoperatively simulate a surgicalprocedure before the procedure is performed intraoperatively. Thesimulation may use one or more physical tools, such as the tool attachedto the manipulator. Alternatively, the simulation may use one or morevirtual tools that are displayed by a display.

As used herein, the term “simulation” refers to the preoperativeperformance of the steps of a surgical workflow corresponding to anintraoperative performance of a surgical procedure. The simulationallows a surgeon to practice the procedure before the procedure actuallytakes place on a real patient. Accordingly, at least some of the stepsof the simulation directly correspond to the steps of the intraoperativeprocedure in some embodiments. As described more fully herein, thesurgeon may adjust a pose of the tool to interact with the preoperativemodel of the patient anatomy during the simulation. For example, thesurgeon may move the tool into contact with the model of the anatomy andmay use the tool to cut away portions of the model of the anatomy orotherwise interact with the model of the anatomy during the simulation.The simulation of the surgical procedure may involve the use of the sametool (or the same type of tool) as will be used in the actualintraoperative procedure, or may involve the use of a virtualrepresentation of the physical tool that will be used in the actualprocedure.

As used herein, the term “movement” of a tool or other object refers toa change in pose of the tool or object over time. The pose of the toolor object includes the position and/or orientation of the tool orobject. Accordingly, the tracking of a tool may include tracking thepose (i.e., the position and/or orientation) of the tool in addition totracking other parameters of the tool.

In embodiments in which a physical tool is used during the simulation ofthe surgical procedure, a surgeon may operate the physical tool toperform the steps of the surgical workflow on a mannequin, a physicalmodel of the patient's anatomy (e.g., a cast mold or model of thepatient's anatomy), or the like. Accordingly, as used herein, the term“physical tool” refers to a surgical tool or other physical toolsimulating operation of a surgical tool that may be physically handledby the surgeon to interact with the patient's anatomy or to interactwith a physical model or mold of an anatomy. For example, in oneembodiment, the physical tool may be a handheld wand or other devicethat is tracked by a camera or other tracking sensor. The tool mayinclude an integrated infrared or other marker that may cooperate withthe tracking sensor to track the pose of the tool over time. When viewedusing a head mounted display (HMD) or the like, a virtual graphicalrepresentation of the surgical tool may be displayed to the user by theHMD. Alternatively, the physical tool may be physically handled by thesurgeon to perform a simulation using a virtual model of an anatomy asdescribed herein.

In embodiments in which a virtual tool is used during the simulation ofthe surgical procedure, the surgeon may provide inputs into a computerto operate the virtual tool to perform the steps of the surgicalworkflow on patient image data, such as a two-dimensional image of thepatient's anatomy or a three-dimensional image or model of the patient'sanatomy. Accordingly, as used herein, the term “virtual tool” refers toa graphical model or image of a surgical tool that is displayed within adisplay device and that may be virtually moved as a result of a surgeonmanipulating an input device such as a mouse, keyboard, touch sensitivescreen, gesture input via computer vision, wand or other device that istracked by a camera or other tracking device, or the like. The virtualtool may be visually displayed as interacting with a virtual model ofthe patient's anatomy in some embodiments.

During the simulation of the surgical procedure using a physical tool, aphysical or virtual model of the patient's anatomy is provided. Thenavigation system tracks the pose of the tool in relation to the modelof the patient's anatomy using a plurality of trackers. As the surgeonperforms the steps of the surgical workflow associated with the surgicalprocedure using the physical tool, a surgical planning program generatesone or more planning parameters to be included within a surgical plan.The planning parameters and the surgical plan are stored within thesurgical planning program when the simulation has completed.

During the simulation of the surgical procedure using a virtual tool,patient image data is displayed on a display. The patient image data maybe preoperatively obtained using MRI, CT, PET, fluoroscopy, and/or anyother imaging modality. As noted above, the patient image data may bedisplayed as one or more two-dimensional images of the patient'sanatomy, or may be displayed as a three-dimensional model of thepatient's anatomy. A virtual representation of the tool is alsodisplayed on the display in relation to the patient image data. As thesurgeon performs the steps of the surgical workflow associated with thesurgical procedure using the virtual tool, the surgical planning programgenerates one or more planning parameters to be included within thesurgical plan. The planning parameters and the surgical plan are storedwithin the surgical planning program when the simulation has completed.

In some embodiments, preoperative simulation may involve a combinationof both physical and virtual techniques. For instance, a physical toolmay be used during the simulation and a computer display, such asaugmented reality eyewear, may project patient image data onto thepatient model as the physical tool interacts with the patient model.

After the simulation has completed, the surgeon may perform the surgicalprocedure intraoperatively on the actual patient using a physical tool.The planning parameters and the surgical plan are loaded from thesurgical planning program. The tool, the patient's anatomy, and one ormore of the planning parameters can be registered by the navigationsystem and are transformed into the coordinate system of the navigationsystem. The planning parameters may then be displayed to the surgeonwhile the surgeon executes the surgical procedure.

The above embodiment is described with the surgeon performing theintraoperative surgical procedure (by either manually operating thesurgical tool or operating the surgical tool semi-autonomously withassistance from the manipulator). However, in an alternative embodiment,the manipulator may autonomously perform the surgical procedure byautonomously operating the surgical tool. In such an embodiment, themanipulator may load the planning parameters into memory and amanipulator controller may automatically cause the manipulator to followthe planning parameters. For example, in an embodiment in which aplanning parameter identifies a path for the surgical tool to follow,the manipulator controller may autonomously control the movement of thesurgical tool to follow the tool path identified in the planningparameter.

Accordingly, the embodiments described herein provide an improvedsurgical experience to surgeons or other health care professionalsinvolved in the surgical procedure. The simulation of the surgicalprocedure enables the surgeon to trial a variety of approaches toperforming the surgical workflow associated with the surgical procedurewithout fear of harming the patient. The simulation program canintelligently identify behavior and/or actions relating to the toolduring the simulation for automatically generating parameters or plansfor intraoperative surgery. The surgeon may then use the parameters orsurgical plan generated as a result of the simulation and may view theparameters intraoperatively while performing the actual surgicalprocedure. As a result, the surgeon may reduce a number of errors duringthe actual surgery and may realize an increased level of confidence inperforming the actual surgical procedure using the embodiments describedherein.

Referring to FIG. 1, a surgical system 10 for treating a patient isillustrated. The surgical system 10 is shown in a surgical setting suchas an operating room of a medical facility. As shown in FIG. 1, thesurgical system 10 may be used to perform an intraoperative surgicalprocedure on a patient. In addition, the surgical system 10, or portionsthereof, may be used to perform a preoperative simulation of thesurgical procedure as described more fully herein.

In the embodiment shown, the surgical system 10 includes a manipulator12 and a navigation system 20. The navigation system 20 is set up totrack movement of various objects in the operating room. Such objectsinclude, for example, a surgical tool 22, a femur F of a patient, and atibia T of the patient. The navigation system 20 tracks these objectsfor purposes of displaying their relative positions and orientations tothe surgeon and, in some cases, for purposes of controlling orconstraining movement of the surgical tool 22 relative to virtualcutting boundaries (not shown) associated with the femur F and tibia T.An example control scheme for the surgical system 10 is shown in FIG. 2.

While the surgical system 10 is illustrated in FIGS. 1-3 as including asurgical robot (i e , manipulator 12) that includes a surgical tool 22attached to an end of the manipulator 12, it should be recognized thatthe surgical system 10 may include one or more manually-operatedsurgical tools 22 instead. For example, the surgical tool 22 may includea hand-held motorized saw, reamer, bur, or other suitable tool that maybe held and manually operated by a surgeon. The following embodimentswill be described with reference to the use of the manipulator 12 withthe understanding that the embodiments may also apply to the use of amanually-operated tool 22 with appropriate modifications. In addition,the following embodiments describe the use of the surgical system 10 inperforming a procedure in which material is removed from a femur Fand/or a tibia T of a patient. However, it should be recognized that thesurgical system 10 may be used to perform any suitable procedure inwhich material is removed from any suitable portion of a patient'sanatomy (e.g., an osteotomy), material is added to any suitable portionof the patient's anatomy (e.g., an implant, graft, etc.), and/or inwhich any other control over a surgical tool is desired.

The navigation system 20 includes one or more computer cart assemblies24 that houses one or more navigation controllers 26. A navigationinterface is in operative communication with the navigation controller26. The navigation interface includes one or more displays 28, 29adjustably mounted to the computer cart assembly 24 or mounted toseparate carts as shown. Input devices I such as a keyboard and mousecan be used to input information into the navigation controller 26 orotherwise select/control certain aspects of the navigation controller26. Other input devices I are contemplated including a touch screen, amicrophone for voice-activation input, an optical sensor for gestureinput, and the like.

A surgical navigation localizer 34 communicates with the navigationcontroller 26. In the embodiment shown, the localizer 34 is an opticallocalizer and includes a camera unit 36. In other embodiments, thelocalizer 34 employs other modalities for tracking, e.g., radiofrequency (RF), ultrasonic, electromagnetic, inertial, and the like. Thecamera unit 36 has a housing 38 comprising an outer casing that housesone or more optical position sensors 40. In some embodiments at leasttwo optical sensors 40 are employed, preferably three or four. Theoptical sensors 40 may be separate charge-coupled devices (CCD). In oneembodiment three, one-dimensional CCDs are employed. Two-dimensional orthree-dimensional sensors could also be employed. It should beappreciated that in other embodiments, separate camera units, each witha separate CCD, or two or more CCDs, could also be arranged around theoperating room. The CCDs detect light signals, such as infrared (IR)signals.

Camera unit 36 is mounted on an adjustable arm to position the opticalsensors 40 with a field-of-view of the below discussed trackers that,ideally, is free from obstructions. In some embodiments the camera unit36 is adjustable in at least one degree of freedom by rotating about arotational joint. In other embodiments, the camera unit 36 is adjustableabout two or more degrees of freedom.

The camera unit 36 includes a camera controller 42 in communication withthe optical sensors 40 to receive signals from the optical sensors 40.The camera controller 42 communicates with the navigation controller 26through either a wired or wireless connection (not shown). One suchconnection may be an IEEE 1394 interface. Additionally or alternatively,the connection may use a company-specific protocol. In otherembodiments, the optical sensors 40 communicate directly with thenavigation controller 26.

Position and orientation signals and/or data are transmitted to thenavigation controller 26 for purposes of tracking objects. The computercart assembly 24, display 28, and camera unit 36 may be like thosedescribed in U.S. Pat. No. 7,725,162 to Malackowski, et al. issued onMay 25, 2010, entitled “Surgery System,” the disclosure of which ishereby incorporated by reference.

The navigation controller 26 can be a personal computer or laptopcomputer. Navigation controller 26 includes the displays 28, 29, centralprocessing unit (CPU) and/or other processors, memory (not shown), andstorage (not shown). The navigation controller 26 is loaded withsoftware as described below. The software converts the signals receivedfrom the camera unit 36 into data representative of the position andorientation of the objects being tracked.

Navigation system 20 is operable with a plurality of tracking devices44, 46, 48, also referred to herein as trackers. In the illustratedembodiment, one tracker 44 is firmly affixed to the femur F of thepatient and another tracker 46 is firmly affixed to the tibia T of thepatient. Trackers 44, 46 are firmly affixed to sections of bone in anembodiment. For example, trackers 44, 46 may be attached to the femur Fand tibia T in the manner shown in U.S. Pat. No. 7,725,162 toMalackowski, et al. issued on May 25, 2010, entitled “Surgery System,”the disclosure of which is hereby incorporated by reference. Trackers44, 46 may also be mounted like those shown in U.S. patent applicationSer. No. 14/156,856, filed on Jan. 16, 2014, entitled, “NavigationSystems and Methods for Indicating and Reducing Line-of-Sight Errors,”hereby incorporated by reference herein. In additional embodiments, atracker (not shown) may be attached to the patella to track a positionand orientation of the patella. In yet further embodiments, the trackers44, 46 may be mounted to other tissue types or parts of the anatomy.

A tool tracker 48 is shown coupled to the manipulator 12. In otherembodiments, a base tracker (not shown) may be substituted for the tooltracker 48, for example, in embodiments in which a hand-held tool 22 isused. The tool tracker 48 may be integrated into the surgical tool 22during manufacture or may be separately mounted to the surgical tool 22(or to an end effector attached to the manipulator 12 of which thesurgical tool 22 forms a part) in preparation for surgical procedures.The working end of the surgical tool 22, which is being tracked byvirtue of the tool tracker 48, may be referred to herein as an energyapplicator, and may be a rotating bur, electrical ablation device,probe, or the like.

In the embodiment shown, the surgical tool 22 is attached to themanipulator 12. Such an arrangement is shown in U.S. Pat. No. 9,119,655,entitled, “Surgical Manipulator Capable of Controlling a SurgicalInstrument in Multiple Modes,” the disclosure of which is herebyincorporated by reference.

The optical sensors 40 of the localizer 34 receive light signals fromthe trackers 44, 46, 48. In the illustrated embodiment, the trackers 44,46, 48 are passive trackers. In this embodiment, each tracker 44, 46, 48has at least three passive tracking elements or markers (e.g.,reflectors) for transmitting light signals (e.g., reflecting lightemitted from the camera unit 36) to the optical sensors 40. In otherembodiments, active tracking markers can be employed. The active markerscan be, for example, light emitting diodes transmitting light, such asinfrared light. Active and passive arrangements are possible.

The navigation controller 26 includes a navigation processor. It shouldbe understood that the navigation processor could include one or moreprocessors to control operation of the navigation controller 26. Theprocessors can be any type of microprocessor or multi-processor system.The term processor is not intended to limit the scope of any embodimentto a single processor.

The camera unit 36 receives optical signals from the trackers 44, 46, 48and outputs to the navigation controller 26 signals relating to theposition of the tracking markers of the trackers 44, 46, 48 relative tothe localizer 34. Based on the received optical signals, navigationcontroller 26 generates data indicating the relative positions andorientations of the trackers 44, 46, 48 relative to the localizer 34. Inone version, the navigation controller 26 uses well known triangulationmethods for determining position data.

Prior to the start of the surgical procedure, additional data are loadedinto the navigation controller 26. Based on the position and orientationof the trackers 44, 46, 48 and the previously loaded data, navigationcontroller 26 determines the position of the working end of the surgicaltool 22 (e.g., the centroid of a surgical bur) and/or the orientation ofthe surgical tool 22 relative to the tissue against which the workingend is to be applied. In some embodiments, the navigation controller 26forwards these data to a manipulator controller 54. The manipulatorcontroller 54 can then use the data to control the manipulator 12. Thiscontrol can be like that described in U.S. Patent No. 9,119,655,entitled, “Surgical Manipulator Capable of Controlling a SurgicalInstrument in Multiple Modes,” or like that described in U.S. Pat. No.8,010,180, entitled, “Haptic Guidance System and Method”, thedisclosures of which are hereby incorporated by reference.

In one embodiment, the manipulator 12 is controlled to stay within apreoperatively defined virtual boundary (sometimes referred to as astereotactic boundary) set by the surgeon or others (not shown). Thevirtual or stereotactic boundary may be a virtual cutting boundary whichdefines the material of the anatomy (e.g., the femur F and tibia T) tobe removed by the surgical tool 22. More specifically, each of the femurF and tibia T has a target volume of material that is to be removed bythe working end of the surgical tool 22. The target volumes are definedby one or more virtual cutting boundaries. The virtual cuttingboundaries define the surfaces of the bone that should remain after theprocedure. The navigation system 20 tracks and controls the surgicaltool 22 to ensure that the working end, e.g., the surgical bur, onlyremoves the target volume of material and does not extend beyond thevirtual cutting boundary, as disclosed in U.S. Pat. No. 9,119,655,entitled, “Surgical Manipulator Capable of Controlling a SurgicalInstrument in Multiple Modes,” the disclosure of which is herebyincorporated by reference, or as disclosed in U.S. Pat. No. 8,010,180,entitled, “Haptic Guidance System and Method”, the disclosure of whichis hereby incorporated by reference.

The virtual cutting boundary may be defined within a virtual model ofthe anatomy (e.g., the femur F and tibia T), or separately from thevirtual model. The virtual cutting boundary may be represented as a meshsurface, constructive solid geometry (CSG), voxels, or using otherboundary representation techniques. The surgical tool 22 may be used tocut away material from the femur F and tibia T to receive an implant.The surgical implants may include unicompartmental, bicompartmental, ortotal knee implants as shown in U.S. Pat. No. 9,381,085, entitled,“Prosthetic Implant and Method of Implantation,” the disclosure of whichis hereby incorporated by reference. Other implants, such as hipimplants, shoulder implants, spine implants, and the like are alsocontemplated. The focus of the description on knee implants is providedas one example. These concepts can be equally applied to other types ofsurgical procedures, including those performed without placing implants.

The navigation controller 26 also generates image signals that indicatethe relative position of the working end to the tissue. These imagesignals are applied to the displays 28, 29. The displays 28, 29, basedon these signals, generate images that allow the surgeon and staff toview the relative position of the working end to the surgical site. Thedisplays, 28, 29, as discussed above, may include a touch screen orother input/output device that allows entry of commands

Referring to FIG. 3, tracking of objects is generally conducted withreference to a localizer coordinate system LCLZ. The localizercoordinate system has an origin and an orientation (a set of x, y, and zaxes). During the procedure one goal is to keep the localizer coordinatesystem LCLZ in a known position. An accelerometer (not shown) mounted tothe localizer 34 may be used to track sudden or unexpected movement ofthe localizer coordinate system LCLZ, as may occur when the localizer 34is inadvertently bumped by surgical personnel.

Each tracker 44, 46, 48 and object being tracked also has its owncoordinate system separate from the localizer coordinate system LCLZ.Components of the navigation system 20 that have their own coordinatesystems are the bone trackers 44, 46 (only one of which is shown in FIG.3) and the base tracker 48. These coordinate systems are represented as,respectively, bone tracker coordinate systems BTRK1, BTRK2 (only BTRK1shown), and base tracker coordinate system BATR.

Navigation system 20 monitors the positions of the femur F and tibia Tof the patient by monitoring the position of bone trackers 44, 46 firmlyattached to bone. Femur coordinate system is FBONE and tibia coordinatesystem is TBONE, which are the coordinate systems of the bones to whichthe bone trackers 44, 46 are firmly attached.

Prior to the start of the intraoperative procedure, preoperative imagesof the femur F and tibia T are generated (or of other portions of theanatomy in other embodiments). The preoperative images are stored astwo-dimensional or three-dimensional patient image data in acomputer-readable storage device, such as memory within the navigationsystem 20. The patient image data may be based on MRI scans,radiological scans or computed tomography (CT) scans of the patient'sanatomy. The patient image data may then be used to generatetwo-dimensional images or three-dimensional models of the patient'sanatomy. A simulation of the procedure may then be performed, forexample, as described more fully herein with reference to FIGS. 5A-B orFIG. 6.

In preparation for the intraoperative procedure, the images orthree-dimensional models developed from the image data are mapped to thefemur coordinate system FBONE and tibia coordinate system TBONE (seetransform T11). One of these models is shown in FIG. 3 with modelcoordinate system MODEL2. These images/models are fixed in the femurcoordinate system FBONE and tibia coordinate system TBONE. As analternative to taking preoperative images, plans for treatment can bedeveloped in the operating room (OR) from kinematic studies, bonetracing, and other methods. The models described herein may berepresented by mesh surfaces, constructive solid geometry (CSG), voxels,or using other model constructs.

During an initial phase of the intraoperative procedure, the bonetrackers 44, 46 are coupled to the bones of the patient. The pose(position and orientation) of coordinate systems FBONE and TBONE aremapped to coordinate systems BTRK1 and BTRK2, respectively (seetransform T5). In one embodiment, a pointer instrument 252 (TLTK), suchas disclosed in U.S. Pat. No. 7,725,162 to Malackowski, et al., herebyincorporated by reference, having its own tracker, may be used toregister the femur coordinate system FBONE and tibia coordinate systemTBONE to the bone tracker coordinate systems BTRK1 and BTRK2,respectively. Given the fixed relationship between the bones and theirbone trackers 44, 46, positions and orientations of the femur F andtibia T in the femur coordinate system FBONE and tibia coordinate systemTBONE can be transformed to the bone tracker coordinate systems BTRK1and BTRK2 so the camera unit 36 is able to track the femur F and tibia Tby tracking the bone trackers 44, 46. These pose-describing data arestored in memory integral with both manipulator controller 54 andnavigation controller 26.

The working end of the surgical tool 22 has its own coordinate system.In some embodiments, the surgical tool 22 comprises a handpiece and anaccessory that is removably coupled to the handpiece. The accessory maybe referred to as the energy applicator and may comprise a bur, anelectrosurgical tip, an ultrasonic tip, or the like. Thus, the workingend of the surgical tool 22 may comprise the energy applicator. Thecoordinate system of the surgical tool 22 is referenced herein ascoordinate system EAPP. The origin of the coordinate system EAPP mayrepresent a centroid of a surgical cutting bur, for example. In otherembodiments, the accessory may simply comprise a probe or other surgicaltool with the origin of the coordinate system EAPP being a tip of theprobe. The pose of coordinate system EAPP is registered to the pose ofbase tracker coordinate system BATR before the procedure begins (seetransforms T1, T2, T3). Accordingly, the poses of these coordinatesystems EAPP, BATR relative to each other are determined. Thepose-describing data are stored in memory integral with both manipulatorcontroller 54 and navigation controller 26.

Referring to FIG. 2, a localization engine 100 is a software module thatcan be considered part of the navigation system 20. Components of thelocalization engine 100 run on navigation controller 26. In someembodiments, the localization engine 100 may run on the manipulatorcontroller 54.

Localization engine 100 receives as inputs the optically-based signalsfrom the camera controller 42 and, in some embodiments, non-opticallybased signals from the tracker controller. Based on these signals,localization engine 100 determines the pose of the bone trackercoordinate systems BTRK1 and BTRK2 in the localizer coordinate systemLCLZ (see transform T6). Based on the same signals received for the basetracker 48, the localization engine 100 determines the pose of the basetracker coordinate system BATR in the localizer coordinate system LCLZ(see transform T1).

The localization engine 100 forwards the signals representative of theposes of trackers 44, 46, 48 to a coordinate transformer 102. Coordinatetransformer 102 is a navigation system software module that runs onnavigation controller 26. Coordinate transformer 102 references the datathat defines the relationship between the preoperative images of thepatient and the bone trackers 44, 46. Coordinate transformer 102 alsostores the data indicating the pose of the working end of the surgicaltool 22 relative to the base tracker 48.

During the procedure, the coordinate transformer 102 receives the dataindicating the relative poses of the trackers 44, 46, 48 to thelocalizer 34. Based on these data, the previously loaded data, and thebelow-described encoder data from the manipulator 12, the coordinatetransformer 102 generates data indicating the relative positions andorientations of the coordinate system EAPP and the bone coordinatesystems, FBONE and TBONE.

As a result, coordinate transformer 102 generates data indicating theposition and orientation of the working end of the surgical tool 22relative to the tissue (e.g., bone) against which the working end isapplied Image signals representative of these data are forwarded todisplays 28, 29 enabling the surgeon and staff to view this information.In certain embodiments, other signals representative of these data canbe forwarded to the manipulator controller 54 to guide the manipulator12 and corresponding movement of the surgical tool 22.

In this example, the surgical tool 22 forms part of the end effector ofthe manipulator 12. The manipulator 12 has a base 57, a plurality oflinks 58 extending from the base 57, and a plurality of active joints(not numbered) for moving the surgical tool 22 with respect to the base57. The manipulator 12 has the ability to operate in a manual mode or asemi-autonomous mode in which the surgical tool 22 is moved along apredefined tool path, as described in U.S. Pat. No. 9,119,655, entitled,“Surgical Manipulator Capable of Controlling a Surgical Instrument inMultiple Modes,” the disclosure of which is hereby incorporated byreference, or the manipulator 12 may be configured to move in the mannerdescribed in U.S. Pat. No. 8,010,180, entitled, “Haptic Guidance Systemand Method”, the disclosure of which is hereby incorporated byreference.

The manipulator controller 54 can use the position and orientation dataof the surgical tool 22 and the patient's anatomy to control themanipulator 12 as described in U.S. Pat. No. 9,119,655, entitled,“Surgical Manipulator Capable of Controlling a Surgical Instrument inMultiple Modes,” the disclosure of which is hereby incorporated byreference, or to control the manipulator 12 as described in U.S. Pat.No. 8,010,180, entitled, “Haptic Guidance System and Method”, thedisclosure of which is hereby incorporated by reference.

The manipulator controller 54 may have a central processing unit (CPU)and/or other manipulator processors, memory (not shown), and storage(not shown). The manipulator controller 54, also referred to as amanipulator computer, is loaded with software as described below. Themanipulator processors could include one or more processors to controloperation of the manipulator 12. The processors can be any type ofmicroprocessor or multi-processor system. The term processor is notintended to limit any embodiment to a single processor.

A plurality of position sensors S are associated with the plurality oflinks 58 of the manipulator 12. In one embodiment, the position sensorsS are encoders. The position sensors S may be any suitable type ofencoder, such as rotary encoders. Each position sensor S is associatedwith a joint actuator, such as a joint motor M. Each position sensor Sis a sensor that monitors the angular position of one of six motordriven links 58 of the manipulator 12 with which the position sensor Sis associated. Multiple position sensors S may be associated with eachjoint of the manipulator 12 in some embodiments. The manipulator 12 maybe in the form of a conventional robot or other conventional machiningapparatus, and thus the components thereof shall not be described indetail.

In some modes, the manipulator controller 54 determines the desiredlocation to which the surgical tool 22 should be moved. Based on thisdetermination, and information relating to the current location (e.g.,pose) of the surgical tool 22, the manipulator controller 54 determinesthe extent to which each of the plurality of links 58 needs to be movedin order to reposition the surgical tool 22 from the current location tothe desired location. The data regarding where the plurality of links 58are to be positioned is forwarded to joint motor controllers JMCs thatcontrol the joints of the manipulator 12 to move the plurality of links58 and thereby move the surgical tool 22 from the current location tothe desired location. In other modes, the manipulator 12 is capable ofbeing manipulated as described in U.S. Pat. No. 8,010,180, entitled,“Haptic Guidance System and Method”, the disclosure of which is herebyincorporated by reference, in which case the actuators are controlled bythe manipulator controller 54 to provide gravity compensation to preventthe surgical tool 22 from lowering due to gravity and/or to activate inresponse to a user attempting to place the working end of the surgicaltool 22 beyond a virtual boundary.

In order to determine the current location of the surgical tool 22, datafrom the position sensors S is used to determine measured joint angles.The measured joint angles of the joints are forwarded to a forwardkinematics module, as known in the art. Based on the measured jointangles and preloaded data, the forward kinematics module determines thepose of the surgical tool 22 in a manipulator coordinate system MNPL(see transform T3 in FIG. 3). The preloaded data are data that definethe geometry of the plurality of links 58 and joints. With thisencoder-based data, the manipulator controller 54 and/or navigationcontroller 26 can transform coordinates from the localizer coordinatesystem LCLZ into the manipulator coordinate system MNPL, vice versa, orcan transform coordinates from one coordinate system into any othercoordinate system described herein using conventional transformationtechniques. In many cases, the coordinates of interest associated withthe surgical tool 22 (e.g., the tool center point or TCP), the virtualboundaries, and the tissue being treated, are transformed into a commoncoordinate system for purposes of relative tracking and display.

In the embodiment shown in FIG. 3, transforms T1-T6 are utilized totransform all relevant coordinates into the femur coordinate systemFBONE so that the position and/or orientation of the surgical tool 22can be tracked relative to the position and orientation of the femur(e.g., the femur model) and/or the position and orientation of thevolume of material to be treated by the surgical tool 22 (e.g., acut-volume model: see transform T10). The relative positions and/ororientations of these objects can also be represented on the displays28, 29 to enhance the user's visualization before, during, and/or aftersurgery.

Referring back to FIGS. 1 and 2, a head-mounted display (HMD) 200 may beemployed to enhance visualization before, during, and/or after surgery.The HMD 200 can be used to visualize the same objects previouslydescribed as being visualized on the displays 28, 29, and can also beused to visualize other objects, features, instructions, warnings, etc.The HMD 200 can be used to assist with visualization of the volume ofmaterial to be cut from the patient, to help visualize the size ofimplants and/or to place implants for the patient, to assist withregistration and calibration of objects being tracked via the navigationsystem 20, to see instructions and/or warnings, among other uses, asdescribed further below.

The HMD 200 may be a mixed reality HMD that overlays computer-generatedimages onto objects viewed in the real world. Thus, in the embodimentdescribed herein, the HMD provides a computational holographic display.Other types of mixed reality HMDs may also be used such as those thatoverlay computer-generated images onto video images of the real world.In other embodiments, the HMD 200 may include an augmented realitydevice, a virtual reality device, or a holographic projection device.The HMD 200 may comprise a cathode ray tube display, liquid crystaldisplay, liquid crystal on silicon display, or organic light-emittingdiode display. The HMD 200 may comprise see-through techniques like thatdescribed herein comprising a diffractive waveguide, holographicwaveguide, polarized waveguide, reflective waveguide, or switchablewaveguide.

The HMD 200 includes a head-mountable structure 202, which may be in theform of an eyeglass and may include additional headbands or supports tohold the HMD 200 on the user's head. In other embodiments, the HMD 200may be integrated into a helmet or other structure worn on the user'shead, neck, and/or shoulders.

The HMD 200 has visor 204 and a lens/waveguide arrangement 208. Thelens/waveguide arrangement 208 is configured to be located in front ofthe user's eyes when the HMD is placed on the user's head. The waveguidetransmits the computer-generated images to the user's eyes while at thesame time, real images are seen through the waveguide (it beingtransparent) such that the user sees mixed reality (virtual and real).

An HMD controller 210 comprises an image generator 206 that generatesthe computer-generated images (also referred to as virtual images) andthat transmits those images to the user through the lens/waveguidearrangement 208. The HMD controller 210 controls the transmission of thecomputer-generated images to the lens/waveguide arrangement 208 of theHMD 200. The HMD controller 210 may be a separate computer, locatedremotely from the support structure 202 of the HMD 200, or may beintegrated into the support structure 202 of the HMD 200. The HMDcontroller 210 may be a laptop computer, desktop computer,microcontroller, or the like with memory, one or more processors (e.g.,multi-core processors), input devices I, output devices (fixed displayin addition to HMD 200), storage capability, etc.

The HMD 200 comprises a plurality of tracking sensors 212 that are incommunication with the HMD controller 210. In some cases, the trackingsensors 212 are provided to establish a global coordinate system for theHMD 200, also referred to as an HMD coordinate system. The HMDcoordinate system is established by these tracking sensors 212, whichmay comprise CMOS sensors or other sensor types, in some cases combinedwith IR depth sensors, to layout the space surrounding the HMD 200, suchas using structure-from-motion techniques or the like. In oneembodiment, four tracking sensors 212 are employed.

The HMD 200 also comprises a photo/video camera 214 in communicationwith the HMD controller 210. The camera 214 may be used to obtainphotographic or video images 214 with the HMD 200, which can be usefulin identifying objects or markers attached to objects, as will bedescribed further below.

The HMD 200 further comprises an inertial measurement unit IMU 216 incommunication with the HMD controller 210. The IMU 216 may comprise oneor more 3-D accelerometers, 3-D gyroscopes, and the like to assist withdetermining a position and/or orientation of the HMD 200 in the HMDcoordinate system or to assist with tracking relative to othercoordinate systems. The HMD 200 may also comprise an infrared motionsensor 217 to recognize gesture commands from the user. Other types ofgesture sensors are also contemplated. The motion sensor 217 may bearranged to project infrared light or other light in front of the HMD200 so that the motion sensor 217 is able to sense the user's hands,fingers, or other objects for purposes of determining the user's gesturecommand and controlling the HMD 200, HMD controller 210, navigationcontroller 26, and/or manipulator controller 54 accordingly. Gesturecommands can be used for any type of input used by the system 10.

In order for the HMD 200 to be effectively used, the HMD 200 must beregistered to one or more objects used in the operating room, such asthe tissue being treated, the surgical tool 22, the manipulator 12, thetrackers 44, 46, 48, the localizer 34, and/or the like. In oneembodiment, the HMD coordinate system is a global coordinate system(e.g., a coordinate system of the fixed surroundings as shown in FIG.3). In this case, a local coordinate system LOCAL is associated with theHMD 200 to move with the HMD 200 so that the HMD 200 is always in aknown position and orientation in the HMD coordinate system. The HMD 200utilizes the four tracking sensors 212 to map the surroundings andestablish the HMD coordinate system. The HMD 200 then utilizes thecamera 214 to find objects in the HMD coordinate system. In someembodiments, the HMD 200 uses the camera 214 to capture video images ofmarkers attached to the objects and then determines the location of themarkers in the local coordinate system LOCAL of the HMD 200 using motiontracking techniques and then converts (transforms) those coordinates tothe HMD coordinate system.

In another embodiment, a separate HMD tracker 218 (see FIG. 3), similarto the trackers 44, 46, 48, could be mounted to the HMD 200 (e.g., fixedto the support structure 202). The HMD tracker 218 would have its ownHMD tracker coordinate system HMDTRK that is in a knownposition/orientation relative to the local coordinate system LOCAL orcould be calibrated to the local coordinate system LOCAL usingconventional calibration techniques. In this embodiment, the localcoordinate system LOCAL becomes the HMD coordinate system and thetransforms T7 and T8 would instead originate therefrom. The localizer 34could then be used to track movement of the HMD 200 via the HMD tracker218 and transformations could then easily be calculated to transformcoordinates in the local coordinate system LOCAL to the localizercoordinate system LCLZ, the femur coordinate system FBONE, themanipulator coordinate system MNPL, or other coordinate system.

Referring back to FIG. 3, a registration device 220 may be provided witha plurality of registration markers 224 (shown in FIG. 1) to facilitateregistering the HMD 200 to the localizer coordinate system LCLZ. The HMD200 locates the registration markers 224 on the registration device 220in the HMD coordinate system via the camera 214 thereby allowing the HMDcontroller 210 to create a transform T7 from the registration coordinatesystem RCS to the HMD coordinate system. The HMD controller 210 thenneeds to determine where the localizer coordinate system LCLZ is withrespect to the HMD coordinate system so that the HMD controller 210 cangenerate images having a relationship to objects in the localizercoordinate system LCLZ or other coordinate system.

During use, for example, the localizer 34 and/or the navigationcontroller 26 sends data on an object (e.g., the cut volume model) tothe HMD 200 so that the HMD 200 knows where the object is in the HMDcoordinate system and can display an appropriate image in the HMDcoordinate system. In embodiments in which the femur cut volume is to bevisualized by the HMD 200, the localizer 34 and/or navigation controller26 needs three transforms to get the femur cut volume data to thelocalizer coordinate system LCLZ, T9 to transform the femur cut volumecoordinate system MODEL1 to the femur coordinate system FBONE, T5 totransform the femur coordinate system FBONE to the bone trackercoordinate system BTRK1, and T6 to transform the bone tracker coordinatesystem BTRK1 to the localizer coordinate system LCLZ. Once registrationis complete, then the HMD 200 can be used to effectively visualizecomputer-generated images in desired locations with respect to anyobjects in the operating room.

II. Patient Specific Preoperative Simulation Techniques

FIG. 4 is a block diagram of a surgical planning program 300 that isstored within a non-transitory computer-readable medium, such as amemory accessible by the surgical system 10. The surgical planningprogram 300 may be used during a preoperative simulation of a surgicalprocedure to generate one or more planning parameters and/or a surgicalplan to facilitate executing the procedure. In one embodiment, thesurgical planning program 300 may be stored as a plurality of softwareinstructions within a non-transitory storage medium of the navigationsystem 20. In such an embodiment, the instructions of the surgicalplanning program 300 may be executed by one or more processors, such asone or more processors of the navigation controller 26. Alternatively,the surgical planning program 300 may be stored in any suitablecomponent or device of surgical system 10 and may be executed by anysuitable processor of the surgical system 10. For example, the surgicalplanning program 300 may be stored within a simulation computer (notshown) that is used to perform or facilitate the simulation of thesurgical procedure.

The surgical planning program 300 may include a planning parametergenerator 302 and a surgical plan generator 304. The planning parametergenerator 302 and the surgical plan generator 304 may be implemented asone or more software modules including instructions that are executableby one or more processors to perform the functions described herein.

The planning parameter generator 302 generates one or more planningparameters 306 during the simulation. In one embodiment, the planningparameters 306 may include one or more of a stereotactic or virtualboundary for the tool 22, a list of tools 22 used, a sequence ofselection of tools 22 used, an amount of time that each tool was used22, a feed rate of tool 22, a cutting speed of tool 22, an oscillationor rotation speed of tool 22, a path of tool 22 as tool 22 is moved(i.e., adjusted in pose) in relation to the anatomy (either virtualanatomy or physical anatomy), a pose of tool 22, a pose of tool 22 astool 22 advances along the path, a force to be applied to tool 22, adamping to be applied to tool 22, a power to be applied to tool 22, anamount of material to be deposited by tool 22, a function to be executedby tool 22, and/or any other suitable parameters. In some embodiments,one or more planning parameters 306 physically modify the operation ofthe physical tool during the intraoperative surgery. For example,planning parameters 306 such as stereotactic boundaries and tool pathsmay be used by the manipulator 12 to constrain the movement of the tool22 to follow the tool path or to prevent the tool 22 from crossing thestereotactic boundary.

As described herein, the surgical planning program 300 may generate theplanning parameters 306 in a variety of ways. In one example, thesurgical planning program 300 may receive inputs associated with one ormore planning parameters 306 from the surgeon. For example, during thesimulation of the surgical procedure, the surgeon may desire to create astereotactic boundary for constraining movement of the tool 22. Forexample, the stereotactic boundary may ensure that the surgeon does notinadvertently operate the tool 22 in an undesired area of the patient'sanatomy. The surgeon may use one or more of the input devices Itovirtually draw or otherwise provide one or more reference points todefine the boundary on the patient image data or on the model of thepatient anatomy. The surgical planning program 300 may receive theinputs and may generate a planning parameter 306 representing thestereotactic boundary defined by the surgeon. In a similar manner, thesurgical planning program 300 may receive other inputs from the surgeonrepresenting planning parameters 306 to be used in the laterintraoperative procedure. The surgical planning program 300 may capturethese parameters 306 entered by the surgeon and may store the parameters306 in the memory of the program 300.

In another embodiment, the surgical planning program 300 mayautomatically generate planning parameters 306 based on capturingmovement of the tool 22 during the simulation. Such planning parameters306 may be based on inputs from the navigation system 20 or from otherdevices or systems of the surgical system. The automatic generation ofthe planning parameters 306 may be accomplished autonomously by thesurgical planning program 300 without input from the surgeon. Forexample, the surgical planning program 300 may receive inputs from thenavigation system 20 corresponding to the tracked position of the tool22 as the tool 22 is moved in relation to a model of an anatomy duringthe simulation. The surgical planning program 300 may generate a toolpath from at least a portion of the tracked positions of the tool 22 andmay store the tool path as a planning parameter 306 in memory. In someinstances, input from the surgeon may be utilized to confirm or modifyautomatically generated planning parameters 306.

It should be recognized that the tool path generated from the trackedpositions of the tool 22 may be more than a mere compilation of the rawposition data received from the navigation system 20. For example,during the simulation, the surgeon may move the tool 22 at irregularintervals as the surgeon contemplates each stage of the workflow. Thus,the tool 22 may be positioned at different portions of the model of theanatomy for varying amounts of time. In one embodiment, the localizer 34may be configured to periodically capture or sample the position of thetool 22 via the tool tracker 48 at regular time intervals. As a result,the raw position data received from the navigation system 20 may includea redundant number of position samples at various locations of the modelof the anatomy. The surgical planning program 300 may remove or condensethis redundant data and may otherwise generate a best-fit line, path, orsurface that corresponds to the overall movement of the tool 22 inrelation to the model of the anatomy. In a similar manner, thenavigation system 20 may provide other inputs to the surgical planningprogram 300, and the surgical planning program 300 may thenautomatically generate respective planning parameters 306 based on theinputs received.

In one embodiment, the generation of the planning parameters 306 may usemachine learning or artificial intelligence techniques to predictivelygenerate one or more planning parameters 306. More specifically, thesurgical planning program 300 may predictively generate one or moreplanning parameters 306 based on the inputs received from the surgicalsystem 10. For example, the surgeon may move the tool 22 along a toolpath that avoids nearby anatomical features during the simulation. Thesurgical planning program 300 may reference preoperatively generatedpatient image data to determine the anatomical features avoided by thesurgeon and may automatically generate a stereotactic boundary at anedge of each anatomical feature avoided by the surgeon. The surgicalplanning program 300 may then store the generated stereotactic boundaryas a planning parameter 306 in memory.

As another example, the surgeon may move the tool 22 in relation to themodel of the anatomy according to several trial tool paths whiledeciding which path to use in performing the simulated procedure. Thesurgical planning program 300 may receive the various tracked movementsof the tool 22 from the navigation system 20 and may determine that thesurgeon has attempted different paths for the tool 22. The surgicalplanning program 300 may use machine learning or artificial intelligencetechniques to model the most efficient tool path to take and mayautomatically generate a planning parameter 306 corresponding to themost efficient tool path.

In another example, the surgical planning program 300 may receive inputsfrom the manipulator controller 54 identifying the rotational speed ofthe tool 22 during various portions of the simulation. For example, therotational speed of the tool 22 may be detected to be 50 revolutions perminute (RPM) at a first portion of the simulation, 20 RPM at a secondportion of the simulation, and 50 RPM at a third portion of thesimulation. The surgical planning program 300 may determine that aceiling for the rotational speed of the tool 22 should be set at 50 RPMand a floor for the rotational speed of the tool 22 should be set at 20RPM. Alternatively, the surgical planning program 300 may determine thatan acceptable window for the rotational speed of the tool 22 is between20 and 50 RPM. The surgical planning program 300 may then generate aplanning parameter 306 for the rotational speed of the tool 22 even ifthe surgeon does not specify any requirement for the speed. Thus, thesurgical planning program 300 may automatically generate planningparameters 306 relating to parameters of the surgical workflow that arenot specifically input or identified by the surgeon. The above describedexample is merely illustrative, and other tools and/or other toolparameters (e.g., power, force, damping, etc.) may be determined and/orgenerated.

It should be recognized that the above-described embodiments ofgenerating planning parameters 306 may be used in conjunction with eachother, rather than solely as alternatives to each other. Thus, thesurgical planning program 300 may receive some planning parameters 306as inputs from the surgeon, may automatically generate other planningparameters 306 based on inputs from the surgical system 10, and maypredictively generate still other planning parameters 306 based oninputs from the surgical system 10. In addition, the surgical planningprogram 300 may receive a portion of a planning parameter 306 as aninput from the surgeon and may automatically generate the remainingportion of the planning parameter 300 based on inputs from the surgicalsystem 10.

As described herein, the surgical plan generator 304 generates asurgical plan 308 for use in the intraoperative procedure. The surgicalplan 308 may include a name of the surgical procedure, a list ofsurgical workflow steps involved in the procedure, planning parameters306, notes 310, and/or any other suitable data.

In one embodiment, one or more notes 310 associated with one or moreplanning parameters 306 may be generated by the surgical planningprogram 300 and may be stored therein. The notes 310 may include, forexample, reminders, warnings, memos, tips, recommendations, and thelike. The foregoing examples are not meant to be limiting, but rather,the notes 310 may include any suitable text or graphic that may bedisplayed to the surgeon during the execution of the intraoperativeprocedure. In other embodiments, the notes 310 may include haptic,audio, or video notifications that may be presented to the surgeonduring the intraoperative procedure.

As one example, a note 310 reminding the surgeon to exercise additionalcare in performing a particularly difficult portion of the surgicalworkflow may be generated. The note 310 may be associated with aparticular portion of the tool path or stereotactic boundary relating tothe difficult portion of the workflow, for example, such that the note310 may be displayed to the surgeon when the surgeon reaches that pointin the tool path or is in the process of removing patient tissue at ornear the stereotactic boundary. In one embodiment, the surgeon may enterthe notes as the surgeon performs the simulation, or before or afterperforming the simulation. The surgeon may operate an input device I,such as a keyboard and/or mouse, to enter the notes. Alternatively, thesurgeon may speak the notes out loud and the system may usevoice-recognition software to convert the spoken words of the surgeoninto notes 310.

In addition, the surgeon may modify the planning parameters 306, thenotes 310, and/or other portions of the surgical plan 308 after theportions of the surgical plan 308 have been generated. For example, thesurgeon may initiate the performance of the surgical workflow during thesimulation, and planning parameters 306 such as the tool path and poseof the tool 22 may be automatically generated by the surgical planningprogram 300. However, in the event the surgeon makes a mistake inoperating the tool 22 or decides to modify the tool path or pose, thesurgeon may “rewind” or “undo” the respective planning parameters 306.In such an example, the surgeon may use one or more input devices I toenter inputs into the surgical planning program 300 to revise the toolpath and/or pose of the tool 22 to conform to a later-executed workflowthat uses a different tool path and/or pose. The revised planningparameters 306, notes 310, and/or surgical plan 308 may be stored in thememory as part of the surgical planning program 300. The foregoing ismerely one non-limiting example, and the surgical planning program 300may enable the surgeon to revise any other planning parameter 306 asdesired. The operation of the surgical planning program 300 is describedin more detail in FIGS. 5 and 6.

FIGS. 5A and 5B are perspective views of a physical model 400 of apatient's anatomy, which a surgeon is using to simulate a surgicalprocedure according to one example. In one embodiment, the simulationmay be performed using the navigation system 20 or another suitablesystem or device of surgical system 10 that executes the surgicalplanning program 300 (shown in FIG. 4). Alternatively, the simulationmay be performed on a separate simulation computer (not shown) thatstores and executes the surgical planning program 300.

In the example shown in FIGS. 5A and 5B, the surgical procedure is apartial knee replacement surgical procedure. However, it should berecognized that any suitable surgical procedure may be simulatedaccording to the present embodiments.

In the embodiment shown in FIGS. 5A and 5B, the surgeon is using aphysical tool 22 to simulate the surgical procedure. Referring to FIG.5A, one of the steps of the surgical workflow associated with thesimulated surgical procedure is the removal of a target volume 402 ofthe patient's femur F. As illustrated in FIGS. 5A and 5B, for simulationpurposes, a cast mold or other physical model of the patient's femur Fis used.

As an initial step in the simulation, the model of the femur F and thetool 22 are registered in the localizer coordinate system LCLZ. Morespecifically, trackers 44, 46, 48 are firmly attached to the femur F,tibia T, and the tool 22, respectively (or to the manipulator 12 thatholds the tool 22 in one embodiment). The localizer 34 registers thetrackers 44, 46, 48 as described above with reference to FIGS. 1-3.

Referring to FIG. 5B, the surgeon may then operate the physical tool 22,moving the tool 22 in relation to the femur F. The movement of the tool22 in relation to the femur defines a tool path 404. In addition, thesurgeon may orient the tool 22 in one or more poses such that the tool22 is oriented in a first pose 406 along a first portion of the toolpath 404 and is oriented in a second pose 408 along a second portion ofthe tool path 404.

As the surgeon moves the tool 22, the navigation system 20 automaticallytracks the movement to create the tool path 404. The navigation system20 also automatically tracks the poses of the tool 22 as it moves alongthe tool path 404. The navigation system 20 provides the trackedmovement and poses of the tool 22 as inputs to the surgical planningprogram 300 which then generates the tool path 404 and sequence of posesof the tool 22 during the simulation. The surgical planning program 300stores planning parameters 306 associated with the tool path 404 and thesequence of poses in memory of the surgical planning program 300. In asimilar manner, the navigation system 20 may provide other inputs to thesurgical planning program 300, such as the sequence of tools 22 selectedand used during the simulation. The surgical planning program 300 maythen automatically generate respective planning parameters 306 based onthe inputs received.

In another example, the surgical planning program 300 may automaticallygenerate a virtual or stereotactic boundary 412 at the outer edge of thetool path 404 or at another suitable location. Alternatively, thesurgeon may define the virtual boundary 412 by entering one or moreinputs representative of the boundary 412 into the surgical planningprogram 300. The surgical planning program 300 may then store theboundary 412 in the memory of the program 300.

As described above, the surgeon may also dictate or otherwise provideone or more notes 310 during the simulation. For example, if the surgeonencounters unexpected difficulty in performing a particular step of thesurgical workflow, the surgeon may enter a note 310 into the surgicalplanning program 300 identifying the difficulty and any tips orreminders of how to successfully complete that step of the workflow. Thenotes 310 may then be associated with the particular step of theworkflow or with a particular portion of the tool path 404, for example.

After the surgeon is finished with the simulation of the surgicalprocedure, the surgeon may enter an input into the surgical planningprogram 300 indicating that the simulation is completed. In response toreceiving the input, the surgical planning program 300 may stopgenerating planning parameters 306 and may generate the surgical plan308 based on the previously generated notes 310 and parameters 306. Thesurgical planning program 300 may then store the completed surgical plan308 in memory for later access during the intraoperative procedure.

While the embodiment described above uses the localizer to track themovement of the tool 22 and the anatomy to determine the tool pathand/or other parameters 306, it should be recognized that other suitablemethods for determining planning parameters 306 may be used. Forexample, in one embodiment, a scanner, optical sensor, or similar device(not shown) may be used to determine the locations of material removedfrom the physical model 400 of the patient's anatomy. More specifically,a scanner, optical sensor, or other suitable device may be used todetermine an initial contour, shape, volume, and/or other similarcharacteristics of the physical model 400 of the anatomy before thesimulation is performed. The characteristics may then be stored inmemory for later comparison. After the surgeon has completed thesimulation, the scanner, optical sensor, or other device may be used todetermine the final contour, shape, volume, and/or other characteristicof the physical model 400. The surgical planning program may thencompare the final characteristic with the initial characteristic todetermine an amount of material removed from the model 400, one or morestereotactic boundaries for the tool 22, and/or any other suitableparameter 306.

FIG. 6 is a perspective view of a virtual model 500 of a patient'sanatomy which a surgeon is using to simulate a surgical procedure. Inone embodiment, the simulation may be performed using the navigationsystem 20 or another suitable system or device of surgical system 10that executes the surgical planning program 300 (shown in FIG. 4).Alternatively, the simulation may be performed on a separate simulationcomputer (not shown) that stores and executes the surgical planningprogram 300.

In the embodiment shown in FIG. 6, the virtual model 500 is displayedwithin a display 501. In one embodiment, the display 501 is one or moreof the displays 28, 29. Alternatively, the display 501 may be part ofHMD 200 or another suitable display. As illustrated in FIG. 6, thevirtual model 500 may be a three-dimensional model or other visualrepresentation of patient image data that was preoperatively acquiredvia MRI, CT, or another suitable imaging modality. Thus, in thisembodiment, the virtual model is a representation of the anatomy of thepatient that will later undergo the surgical operation that is beingsimulated in FIG. 6. While FIG. 6 illustrates a three-dimensional modelof the patient's anatomy, it should be recognized that the patient imagedata may be alternatively displayed as a two-dimensional image of thepatient's anatomy. In the example shown in FIG. 6, the surgicalprocedure is a partial knee replacement surgical procedure. However, itshould be recognized that any suitable surgical procedure may besimulated according to the present embodiments.

In the embodiment shown in FIG. 6, the surgeon is using a virtualrepresentation 502 of a physical tool 22 (hereinafter referred to as avirtual tool 502) to simulate the surgical procedure. Thus, in oneembodiment, the virtual tool 502 is a three-dimensional model of thetool 22 that will later be used in the execution of the intraoperativesurgical procedure that is currently being simulated. Alternatively, thevirtual tool 502 may be displayed as a two-dimensional image of thephysical tool 22 that will later be used to perform the intraoperativeprocedure. The virtual tool 502 may be displayed in relation to thevirtual model 500 of the anatomy within display 501.

In one embodiment, one of the steps of the surgical workflow associatedwith the simulated surgical procedure may be the removal of a targetvolume 504 of the patient's femur F. The surgeon may operate the virtualtool 502 by entering inputs into the surgical planning program 300 usingone or more input devices I of the navigation system 20 representing adesired change in position or pose of the tool 502. In response, thesurgical planning program 300 moves the virtual tool 502 in relation tothe image of the femur F. The movement of the tool 502 in relation tothe femur defines a virtual tool path 506. In addition, the surgeon mayorient the virtual tool 502 in one or more poses in a similar manner asdescribed above.

The surgeon may also operate one or more input devices I to adjust theresolution of the patient image data. For example, the surgeon may“zoom” in or out of the patient image data and may adjust the viewingperspective of the image data to provide a desired viewpoint andresolution to the surgeon.

As the surgeon moves the virtual tool 502, the surgical planning program300 tracks the change in pose of the tool as the tool moves along thetool path. The surgical planning program 300 stores planning parameters306 associated with the tool path 506 and the sequence of poses inmemory of the surgical planning program 300. In a similar manner, thesurgical planning program 300 may automatically generate other planningparameters 306 in a similar manner as described above.

As described above, the surgeon may also dictate or otherwise provideone or more notes 310 during the simulation. The notes 310 may then beassociated with the particular step of the workflow or with a particularportion of the tool path 506, for example.

FIG. 7 is a perspective view of a virtual model 400 of a patient'sanatomy, which a surgeon is using to simulate a surgical procedure usinga physical tool 22 according to one example. In one embodiment, thephysical tool 22 is a surgical tool that is the same as, or similar to,the actual surgical tool that will be used in the later intraoperativeprocedure. In an alternative embodiment, the physical tool 22 may be ahandheld wand or other device that is tracked by a camera (e.g., thecamera unit 36) or other tracking sensor. In the alternative embodiment,the tool may include an integrated infrared or other marker that maycooperate with the tracking sensor to track the pose of the tool overtime. When viewed using a head mounted display (HMD) 200 or the like, avirtual graphical representation of the surgical tool may be displayedto the user by the HMD 200. In one embodiment, the simulation may beperformed using the navigation system 20 or another suitable system ordevice of surgical system 10 that executes the surgical planning program300 (shown in FIG. 4). Alternatively, the simulation may be performed ona separate simulation computer (not shown) that stores and executes thesurgical planning program 300.

In the embodiment shown in FIG. 7, the surgeon is using a physical tool22 to simulate the surgical procedure. One of the steps of the surgicalworkflow associated with the simulated surgical procedure is the removalof a target volume 602 of the patient's femur F. As illustrated in FIG.7, for simulation purposes, a virtual model 400 of a portion of thepatient's femur F is used.

In the embodiment shown in FIG. 7, the virtual model 400 is displayed bya display 604. In one embodiment, the display 604 is part of HMD 200 oranother suitable display. For example, the virtual model 400 may bedisplayed on one or more display screens (e.g., eyeglass lenses) of theHMD 200 as the HMD 200 is worn by the surgeon. As illustrated in FIG. 7,the virtual model 400 may be a three-dimensional model or other visualrepresentation of patient image data that was preoperatively acquiredvia MRI, CT, or another suitable imaging modality. Thus, in thisembodiment, the virtual model 400 is a representation of the anatomy ofthe patient that will later undergo the surgical operation that is beingsimulated in FIG. 7. In the example shown in FIG. 7, the surgicalprocedure is a partial knee replacement surgical procedure. However, itshould be recognized that any suitable surgical procedure may besimulated according to the present embodiments.

As an initial step in the simulation, the virtual model 400 of the femurF and the tool 22 are registered in the localizer coordinate systemLCLZ. More specifically, trackers 48, 218 are coupled to the tool 22 andthe HMD 200 respectively (or to the manipulator 12 that holds the tool22 in one embodiment). The localizer 34 registers the trackers 48, 218as described above with reference to FIGS. 1-3. Once the HMD 200 isregistered within the localizer coordinate system LCLZ, the virtualmodel 400 of the patient's anatomy is likewise registered to thelocalizer coordinate system LCLZ. In one embodiment, the HMD 200 andvirtual model are registered to the localizer coordinate system LCLZ asdescribed in U.S. patent application Ser. No. 15/860,057, entitled“Systems and Methods for Surgical Navigation”, the disclosure of whichis incorporated by reference in its entirety.

The surgeon may then operate the physical tool 22, moving the tool 22 inrelation to the virtual model 400. As the surgeon moves the tool 22, thenavigation system 20 automatically tracks the movement to create a toolpath 606. The navigation system 20 also automatically tracks the posesof the tool 22 as it moves along the tool path 606 in a similar manneras described above with reference to FIGS. 5A and 5B. The navigationsystem 20 provides the tracked movement and poses of the tool 22 asinputs to the surgical planning program 300 which then generates thetool path 606 and sequence of poses of the tool 22 during thesimulation. The surgical planning program 300 stores planning parameters306 associated with the tool path 606 and the sequence of poses inmemory of the surgical planning program 300. In a similar manner, thenavigation system 20 may provide other inputs to the surgical planningprogram 300, such as the sequence of tools 22 selected and used duringthe simulation. The surgical planning program 300 may then automaticallygenerate respective planning parameters 306 based on the inputsreceived. Accordingly, the surgical planning program 300 may generateone or more planning parameters 306 based on the use of the physicaltool 22 in relation to the image representing the portion of thepatient's anatomy as displayed by the display 604.

In another example, the surgical planning program 300 may automaticallygenerate a virtual or stereotactic boundary 608 at the outer edge of thetool path 606 or at another suitable location. Alternatively, thesurgeon may define the virtual boundary 608 by entering one or moreinputs representative of the boundary 608 into the surgical planningprogram 300. The surgical planning program 300 may then store theboundary 608 in the memory of the program 300.

As described above, the surgeon may also dictate or otherwise provideone or more notes 310 during the simulation. For example, if the surgeonencounters unexpected difficulty in performing a particular step of thesurgical workflow, the surgeon may enter a note 310 into the surgicalplanning program 300 identifying the difficulty and any tips orreminders of how to successfully complete that step of the workflow. Thenotes 310 may then be associated with the particular step of theworkflow or with a particular portion of the tool path 606, for example.

After the surgeon is finished with the simulation of the surgicalprocedure, the surgeon may enter an input into the surgical planningprogram 300 indicating that the simulation is completed. In response toreceiving the input, the surgical planning program 300 may stopgenerating planning parameters 306 and may generate the surgical plan308 based on the previously generated notes 310 and parameters 306. Thesurgical planning program 300 may then store the completed surgical plan308 in memory for later access during the intraoperative procedure.

FIG. 8 is a perspective view of a portion of a patient's anatomy whilethe patient is undergoing an intraoperative surgical procedure. In aspecific embodiment, the patient is undergoing the surgical procedurethat the surgeon simulated in accordance with the embodiments describedin FIGS. 5A and 5B or in FIG. 6. Accordingly, the patient is undergoinga partial knee replacement surgical operation in the illustratedembodiment. However, it should be recognized that any suitableintraoperative surgical procedure may be performed that substantiallymatches the surgical procedure that was simulated preoperatively.

As described herein, the intraoperative surgical procedure is executedusing system 10 (shown in FIG. 1). One way that the intraoperativesurgical procedure may be executed is described in U.S. patentapplication Ser. No. 15/860,057, entitled “Systems and Methods forSurgical Navigation”, the disclosure of which is incorporated herein byreference. While the following embodiments are described as beingperformed with the assistance of the manipulator 12 in a semi-autonomousmode of operation, it should be recognized that the embodiments mayalternatively be performed without the use of the manipulator 12. Forexample, the surgeon may operate one or more surgical tools 22 manuallywhile viewing one or more of the planning parameters 306 and/or notes310 on a display, such as a display of HMD 200 and/or a display 28 or 29of the navigation system 20. It should also be recognized that themanipulator 12 may perform the surgical procedure in an autonomous modeof operation in which the manipulator removes the target volume of theanatomy, rather than the surgeon. Notwithstanding the foregoing, thefollowing embodiments will be described with reference to thesemi-autonomous mode of operation unless otherwise specified.

As the surgeon prepares to execute the intraoperative surgicalprocedure, the navigation controller 26 and/or the manipulatorcontroller 54 loads the surgical plan 308 along with the planningparameters 306 and any notes 310 that were generated during thesimulation. As noted above, the planning parameters 306 may include atool path and a pose of the tool 22 at each portion of the tool path.

Since the planning parameters 306 were generated using the surgicalplanning program 300 described above, the planning parameters 306 willneed to be registered to the navigation system of coordinates (i.e., thelocalizer coordinate system LCLZ). To accomplish this, the patient'sanatomy is first registered by affixing trackers 44, 46 to the femur Fand tibia T. The anatomy is then registered by the localizer 34 in amanner described above with reference to FIGS. 1-3. In a similar manner,the tool 22 and/or the manipulator 12 to which the tool 22 is attachedis registered to tracker 48 affixed to the tool 22 or manipulator 12.

If the preoperative simulation was performed using a virtual tool 502 inrelation to a virtual model 500 of the patient's anatomy, the virtualmodel 500 of the patient's anatomy may be matched to the actualpatient's anatomy using a method similar to that described in U.S. Pat.No. 8,675,939, entitled “Registration of Anatomical Data Sets”, thedisclosure of which is hereby incorporated by reference in its entirety.On the other hand, if the preoperative simulation was performed using aphysical tool 22 in relation to a physical model 400 of the patient'sanatomy, the registration data used by the localizer 34 during thesimulation may be correlated to the registration data used by thelocalizer 34 described in the preceding paragraph according to methodsknown in the art. Once the virtual or physical model of the patient'sanatomy used in the simulation is correlated to the intraoperativeregistration data of the localizer 34, the coordinates of the tool pathand other planning parameters 306 may be transformed according to knownmethods. After this, the planning parameters 306 are represented in thelocalizer coordinate system LCLZ.

Once the planning parameters 306 are registered and represented in thelocalizer coordinate system LCLZ, the planning parameters 306, notes310, and/or other portions of the surgical plan 308 may be displayed tothe surgeon during the execution of the intraoperative surgicalprocedure. The planning parameters 306 and other portions of thesurgical plan 308 may be displayed on any suitable display, such as thedisplay 28 or 29, on HMD 200, and/or as a holographic image overlaidonto the patient's anatomy or on a wall or other portion of the system10. The following embodiments are described with reference to thedisplay of the planning parameters 306 on HMD 200 with the understandingthat the embodiment may equally apply to other displays with appropriatemodifications.

In an embodiment in which HMD 200 is used, the HMD 200 must also beregistered to the localizer coordinate system LCLZ. The HMD 200 may beregistered using HMD tracker 218 in a similar manner as described abovewith reference to FIGS. 1-3. In addition, the HMD 200 may be tracked andregistered in a similar manner as described in U.S. patent applicationSer. No. 15/602,261, entitled “Systems and Methods for Identifying andTracking Physical Objects During A Robotic Surgical Procedure”, thedisclosure of which is hereby incorporated by reference in its entirety.

According to one example illustrated in FIG. 8, the HMD 200 can be usedto visually depict the desired tool path for the tool 22 to followduring manual, semi-autonomous, or autonomous movement of the surgicaltool 22. The navigation controller 26 and/or the manipulator controller54 can store the tool path and its associated location data. Thislocation data is transmitted to the HMD controller 210, which generatesa tool path image 702 visually coinciding with the stored tool path suchthat the HMD 200 displays the tool path image 702 to seemingly belocated in the actual bone at the actual locations that the working endof the tool 22 (e.g., the bur) will traverse along the tool path. Theentire tool path can be displayed to the surgeon or only portions of thetool path might be displayed, such as only those portions that have notyet been traversed. In some cases, as the working end of the tool 22successfully follows along segments of the tool path, images associatedwith those segments may disappear and no longer be displayed. In somecases, only a small section of the tool path is displayed ahead of theworking end of the surgical tool 22 to act as a general guide, but notthe entire tool path.

In a similar manner, data representative of the desired pose of thesurgical tool 22 can be provided to the HMD controller 210 so that theHMD controller 210 can generate corresponding pose (i.e., positionand/or orientation) images along one or more segments of the tool pathto visually depict to the surgeon how the surgical tool 22 should bepositioned and/or oriented in the future with respect to the actualanatomy.

According to another embodiment, the HMD 200 may also be used tovisually depict planning parameters 306 representative of virtual orstereotactic boundaries 704 associated with the particular treatment ofthe patient. If the working end of the surgical tool 22 moves nearand/or outside of this virtual boundary 704, haptic feedback may begenerated via the manipulator 12 in the manner described in U.S. PatentNo. 8,010,180, entitled, “Haptic Guidance System and Method”, thedisclosure of which is hereby incorporated by reference. This hapticfeedback indicates to the user that the working end of the surgical tool22 is approaching the virtual boundary 704, has reached the boundary704, or is beyond the boundary 704. In some cases, the boundary 704 isrepresented as a path, such as a trajectory along which the surgicaltool 22 is expected to be oriented and/or along which the surgical tool22 is expected to traverse. In this case, the haptic feedback mayindicate to the user that the surgical tool 22 has been reoriented offthe trajectory or has moved too far along the trajectory.

In a similar manner, any other suitable planning parameter 306 may bedisplayed to the surgeon during the intraoperative surgical procedure onHMD 200. The HMD 200 may also display any notes 310 that were generatedduring the preoperative simulation. In one embodiment, each note 310 isonly displayed when the surgeon reaches the workflow step associatedwith the note 310. In a further embodiment, one or more notes 310 may beaudibly presented to the surgeon by a speaker within the surgical system10 instead of, or in addition to, the note 310 being visually displayedon HMD 200.

Accordingly, as described herein, the display of the planning parameters306, notes 310, and/or other aspects of the surgical plan 308 may assistthe surgeon in performing the intraoperative surgical procedure and mayreduce a number and/or severity of errors that may otherwise occurduring the surgical procedure.

FIG. 8 is a flowchart illustrating a method 800 of performing a surgicalprocedure on a patient. In one embodiment, one or more steps of themethod 800 are embodied as a plurality of instructions that are storedin a non-transitory computer-readable medium and that are executable bya processor. For example, the method 800 may be at least partiallyexecuted by the surgical system 10 when a processor, such as navigationcontroller 26 and/or manipulator controller 54, execute the instructionsassociated with one or more steps. In a further embodiment, at leastsome of the steps of the method 800 may be automatically executed by thesurgical planning program 300 (shown in FIG. 4).

In an embodiment, the method 800 is split into two main flows: apreoperative simulation flow 802 and an intraoperative flow 804. Thepreoperative flow 802 begins with generating 806 preoperative image dataof a patient. For example, preoperative image data may be generatedusing an MRI system, a CT system, a PET system, an x-ray fluoroscopysystem, and/or any other suitable system.

A preoperative simulation of the surgical procedure may then beperformed 808. As part of the preoperative simulation flow 802, one ormore components of the surgical system 10 may need to be registeredusing the localizer 34. For example, if the surgeon is using a physicaltool 22 to simulate the surgical procedure on a physical model of thepatient's anatomy, the physical tool 22 and the physical model of thepatient's anatomy may be registered in a similar manner as describedabove. Alternatively, in an embodiment where the surgeon is operating avirtual representation of a physical tool on a virtual model of thepatient's anatomy during the simulation, no registration of the virtualtool or virtual model of the anatomy is required.

As the surgeon performs the steps of the surgical workflow associatedwith the surgical procedure, at least one planning parameter isautomatically generated 810. For example, the surgical planning program300 may automatically generate a tool path based on the change in poseof the tool by the surgeon as the surgeon performs the steps of thesurgical workflow. One or more notes may also be generated as describedabove.

After the simulation has completed or after the surgeon has completedsimulating the steps of the surgical workflow, the surgical planningprogram may generate 812 a surgical plan based on the planningparameters 306 and/or any notes 310 generated, for example. The surgicalplan 308 and its associated planning parameters 306, notes 310, andother components may then be stored 814 in memory for later retrieval toassist the surgeon in performing the intraoperative procedure.

During the intraoperative flow 804, the surgical plan 308 and itsincluded planning parameters 306 and notes 310 are loaded 816 from thesurgical planning program. The patient's anatomy and the surgical system10 may be registered 818 with the localizer 34 in a similar manner asdescribed above. This step includes firmly attaching trackers 44, 46 tothe patient's anatomy. Alternatively, any suitable number of trackersmay be attached to any suitable portion or portions of the patient'sanatomy. This step also includes firmly attaching tracker 48 to tool 22or manipulator 12, and optionally attaching HMD tracker 218 to HMD 200in embodiments where the HMD 200 is used. The registration of thevarious trackers and the transforms applied to the localizer signalsreceived from each tracker is described above with reference to FIGS.1-3.

The planning parameters 306 and any other portions of the surgical planthat relate to coordinates are also registered 820 with the localizer.The registration of the planning parameters 306 includes applying atransform to the parameters to transform them from the surgical planningprogram coordinate system to the localizer coordinate system LCLZ in asimilar manner as described above. Example parameters 306 that may beregistered include a tool path, a pose of tool 22, and a stereotacticboundary for the tool 22.

The planning parameters 306 are displayed 822 to the surgeon during theintraoperative execution of the surgical procedure. This step mayinclude displaying the parameters 306 on the HMD 200, displaying theparameters 306 on one or more of the displays 28, 29, and/or projectinga holographic image of one or more parameters 306 onto the patient'sanatomy or onto a suitable portion of the surgical system 10.

The surgeon then intra-operatively performs 824 or executes the surgicalprocedure while one or more of the parameters 306 and notes 310 aredisplayed to the surgeon. As the surgeon performs or executes theprocedure, the navigation system 20 tracks the real-time pose of thetool 22 and automatically updates at least one parameter 306 based onthe real-time pose. For example, as the surgeon advances the tool 22along the tool path, the HMD 200 receives the tracking data from thenavigation system 20 and automatically displays the tool path as beingtraversed as the surgeon advances the tool 22 along the tool path. Forexample, the tool path that has already been followed may be displayedin a first color, such as green, and the tool path that the surgeon hasnot yet followed or completed may be displayed in a second color, suchas red. In another example, as the surgeon advances the tool 22 alongthe tool path, the system may determine that the surgeon has completed aparticular stage of the surgical plan and may display a note 310 orother data indicating the completion of the stage. The system may thendisplay a note 310 or other data indicating the next stage of thesurgical plan that must be completed as well as additional notes 310 orother data indicating tips, warnings, instructions, or supplemental datarelating to the upcoming stage. As another example, the system mayautomatically determine, based on the tracked pose of the tool 22, thatthe surgeon has deviated from the tool path or from another planningparameter, or is at risk of doing so, and may display a warning, error,or other notification indicating such.

Several embodiments have been discussed in the foregoing description.However, the embodiments discussed herein are not intended to beexhaustive or limit the invention to any particular form. Theterminology which has been used is intended to be in the nature of wordsof description rather than of limitation. Many modifications andvariations are possible in light of the above teachings and theinvention may be practiced otherwise than as specifically described.

1. A preoperative surgical planning system comprising: a head-mounteddevice comprising one or more controllers and a display that isconfigured to be located in front of the eyes of a wearer of thehead-mounted device, the head-mounted device being configured to receivea control input from the wearer, and the one or more controllers beingconfigured to execute a preoperative surgical simulation with thehead-mounted device wherein the one or more controllers are configuredto: load a virtual tool and a virtual anatomical model for thepreoperative surgical simulation; display the virtual tool and thevirtual anatomical model on the display of the head-mounted device;track the virtual tool relative to the virtual anatomical model in thepreoperative surgical simulation in which the virtual tool is moveablein response to receipt of the control input from the wearer of thehead-mounted device and wherein the virtual tool is configured to removea portion of the virtual anatomical model; automatically generate aplanning parameter based on tracking of the virtual tool relative to thevirtual anatomical model in the preoperative surgical simulation; andstore the generated planning parameter for future retrieval by asurgical system to facilitate intraoperative surgery based on thegenerated planning parameter.
 2. The preoperative surgical planningsystem of claim 1, wherein the virtual anatomical model corresponds to aphysical anatomy to be treated, and wherein the generated planningparameter includes a virtual cutting boundary, the virtual cuttingboundary configured to constrain movement of an intraoperative tool ofthe surgical system from reaching an undesired area of the physicalanatomy.
 3. The preoperative surgical planning system of claim 1,wherein the virtual anatomical model corresponds to a physical anatomyto be treated, and wherein the generated planning parameter includes atool path, the tool path being configured to constrain movement of anintraoperative tool of the surgical system relative to the physicalanatomy.
 4. The preoperative surgical planning system of claim 1,wherein the generated planning parameter includes one or more of: a feedrate of the virtual tool, a cutting speed of the virtual tool, anorientation of the virtual tool, and a sequence of selection of aplurality of virtual tools.
 5. The preoperative surgical planning systemof claim 1, wherein the one or more controllers are configured to:capture a first state of the virtual anatomical model before removal ofthe portion of the virtual anatomical model by the virtual tool; capturea second state of the virtual anatomical model after removal of theportion of the virtual anatomical model by the virtual tool; and comparethe first state and the second state of the virtual anatomical model toautomatically generate the planning parameter.
 6. The preoperativesurgical planning system of claim 5, wherein the virtual anatomicalmodel corresponds to a physical anatomy to be treated, and wherein theone or more controllers are configured to automatically generate theplanning parameter to include planned removal dimensions for thephysical anatomy to be treated, and wherein the planned removaldimensions are derived from comparison of the first state and the secondstate of the virtual anatomical model.
 7. The preoperative surgicalplanning system of claim 1, wherein the display of the head-mounteddevice comprises one or more of: an augmented reality display, a virtualreality display, a mixed reality display, or a holographic display. 8.The preoperative surgical planning system of claim 1, wherein thehead-mounted device comprises a sensing system coupled to the one ormore controllers, the sensing system being configured to sense an actionof the wearer of the head-mounted device and the one or more controllersare configured to move the virtual tool in the preoperative surgicalsimulation in response to receipt of the control input from the sensedaction.
 9. The preoperative surgical planning system of claim 8, whereinthe sensing system comprises one or more of: a computer vision system,an infrared motion sensor, an inertial measurement system, and atrackable hand-held wand.
 10. A method of operating a preoperativesurgical planning system, the preoperative surgical planning systemcomprising a head-mounted device including one or more controllers and adisplay that is configured to be located in front of the eyes of awearer of the head-mounted device, the head-mounted device beingconfigured to receive a control input from the wearer, and the one ormore controllers being configured to execute a preoperative surgicalsimulation with the head-mounted device, wherein the method comprisesthe one or more controllers performing the following steps: loading avirtual tool and a virtual anatomical model for the preoperativesurgical simulation; displaying the virtual tool and the virtualanatomical model on the display of the head-mounted device; tracking thevirtual tool relative to the virtual anatomical model in thepreoperative surgical simulation whereby the virtual tool moves inresponse to receiving the control input from the wearer of thehead-mounted device and wherein the virtual tool is moved for removing aportion of the virtual anatomical model; automatically generating aplanning parameter based on tracking of the virtual tool relative to thevirtual anatomical model in the preoperative surgical simulation; andstoring the generated planning parameter for future retrieval by asurgical system to facilitate intraoperative surgery based on thegenerated planning parameter.
 11. The method of claim 10, wherein thevirtual anatomical model corresponds to a physical anatomy to betreated, and further comprising automatically generating the planningparameter to be a virtual cutting boundary, the virtual cutting boundaryconfigured to constrain movement of an intraoperative tool of thesurgical system from removing an undesired area of the physical anatomy.12. The method of claim 10, wherein the virtual anatomical modelcorresponds to a physical anatomy to be treated, and further comprisingautomatically generating the planning parameter to be a tool path, thetool path being configured to constrain movement of an intraoperativetool of the surgical system relative to the physical anatomy.
 13. Themethod of claim 10, further comprising automatically generating theplanning parameter to be one or more of: a feed rate of the virtualtool, a cutting speed of the virtual tool, an orientation of the virtualtool, and a sequence of selection of a plurality of virtual tools. 14.The method of claim 10, comprising: capturing a first state of thevirtual anatomical model before removal of the portion of the virtualanatomical model by the virtual tool; capturing a second state of thevirtual anatomical model after removal of the portion of the virtualanatomical model by the virtual tool; and comparing the first state andthe second state of the virtual anatomical model for automaticallygenerating the planning parameter.
 15. The method of claim 10,comprising utilizing machine learning or artificial intelligence forpredictively generating the planning parameter.
 16. A system comprising:a preoperative surgical planning system comprising: a head-mounteddevice comprising one or more controllers and a display that isconfigured to be located in front of the eyes of a wearer of thehead-mounted device, the head-mounted device being configured to receivea control input from the wearer, and the one or more controllers beingconfigured to execute a preoperative surgical simulation with thehead-mounted device wherein the one or more controllers are configuredto: load a virtual tool and a virtual anatomical model for thepreoperative surgical simulation, the virtual anatomical modelcorresponding to a physical anatomy to be treated; display the virtualtool and the virtual anatomical model on the display of the head-mounteddevice; track the virtual tool relative to the virtual anatomical modelin the preoperative surgical simulation in which the virtual tool ismoveable in response to receipt of the control input from the wearer ofthe head-mounted device and wherein the virtual tool is configured toremove a portion of the virtual anatomical model; automatically generatea planning parameter based on tracking of the virtual tool relative tothe virtual anatomical model in the preoperative surgical simulation;and store the generated planning parameter; and an intraoperativesurgical system comprising: a robotic manipulator being configured tosupport a surgical tool, the surgical tool being configured to removematerial from the physical anatomy to be treated and the surgical toolcorresponding to the virtual tool used in the preoperative surgicalsimulation, and wherein one or more manipulator controllers are coupledto the robotic manipulator, the one or more manipulator controllersbeing configured to: load the generated planning parameter that wasautomatically generated from the preoperative surgical simulation; andcontrol the robotic manipulator based on the generated planningparameter to remove material from the physical anatomy with the surgicaltool.
 17. The system of claim 16, wherein the generated planningparameter includes a virtual cutting boundary, and wherein theintraoperative surgical system further comprises a tracking systemconfigured to track the surgical tool and the physical anatomy and thetracking system configured to register the virtual cutting boundary tothe physical anatomy, and wherein the one or more manipulatorcontrollers are configured to control the robotic manipulator toconstrain movement of the surgical tool with the virtual cuttingboundary to prevent the surgical tool from removing material from anundesired area of the physical anatomy.
 18. The system of claim 16,wherein the generated planning parameter includes a tool path, andwherein the intraoperative surgical system further comprises a trackingsystem configured to track the surgical tool and the physical anatomyand the tracking system configured to register the tool path to thephysical anatomy, and wherein the one or more manipulator controllersare configured to control the robotic manipulator to constrain movementof the surgical tool along the tool path to remove material from thephysical anatomy.
 19. The system of claim 16, wherein the one or morecontrollers of the head-mounted device are configured to: capture afirst state of the virtual anatomical model before removal of theportion of the virtual anatomical model by the virtual tool; capture asecond state of the virtual anatomical model after removal of theportion of the virtual anatomical model by the virtual tool; and comparethe first state and the second state of the virtual anatomical model toautomatically generate the planning parameter.
 20. The system of claim16, wherein: the one or more controllers of the head-mounted device areconfigured to automatically generate the planning parameter to includeplanned removal dimensions derived from the portion of the virtualanatomical model removed by the virtual tool in the preoperativesurgical simulation; and the intraoperative surgical system furthercomprises a tracking system configured to track the surgical tool andthe physical anatomy and the tracking system configured to register theplanned removal dimensions to the physical anatomy, and wherein the oneor more manipulator controllers are configured to control the roboticmanipulator to remove material from the physical anatomy with thesurgical tool according to the planned removal dimensions.